On the Applicability of Zipf's Law in Chinese Word Frequency Distribution

Zipf's Law uncovers the relationship between word frequency and its rank. This paper addresses applicability of Zipf's Law in Chinese word frequency distribution. The previous studies on Zipf’s law in Chinese were primarily based on raw corpus, without word segmentation, hence there are obvious limitations. This study investigates the topic in several large-scale POS-tagged Chinese corpora. The results of these experiments prove that word frequency distribution in Chinese exhibits Zipf’s law. The paper further examined the distribution of low frequency word in Chinese corpus, which is estimated by Zipf’s law as the majority part of a corpus word list. The result also supports the argument since low frequency words constitute over half of the corpus word occurrences. It indicates that data sparse in statistical approaches could not be magnificently reduced by expanding the corpus scale.